Methods for Sensitive Questions
in Surveys

 

Gustavo Diaz
Postdoctoral Fellow
Department of Political Science
McMaster University
gustavodiaz.org

# Packages
library(tidyverse)
Warning: package 'tidyverse' was built under R version 4.2.3
Warning: package 'ggplot2' was built under R version 4.2.3
Warning: package 'tibble' was built under R version 4.2.3
Warning: package 'readr' was built under R version 4.2.3
Warning: package 'dplyr' was built under R version 4.2.3
Warning: package 'lubridate' was built under R version 4.2.3
── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
✔ dplyr     1.1.1     ✔ readr     2.1.4
✔ forcats   1.0.0     ✔ stringr   1.5.0
✔ ggplot2   3.4.2     ✔ tibble    3.2.1
✔ lubridate 1.9.2     ✔ tidyr     1.3.0
✔ purrr     1.0.1     
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag()    masks stats::lag()
ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors
library(DeclareDesign)
Loading required package: randomizr
Loading required package: fabricatr
Loading required package: estimatr

Attaching package: 'DeclareDesign'

The following object is masked from 'package:dplyr':

    vars

The following object is masked from 'package:ggplot2':

    vars
library(kableExtra)

Attaching package: 'kableExtra'

The following object is masked from 'package:dplyr':

    group_rows
# ggplot global options
theme_set(theme_gray(base_size = 20))
# Data prep
load("attn_rep.RData")

cali = dle

remove(dle)

# Analysis
a = cali %>% 
  split(.$experiment) %>% 
  map(~difference_in_means(listA ~ trt_A, data = .)) %>% 
  map(tidy) %>% 
  bind_rows(.id = "experiment")

b = cali %>% 
  split(.$experiment) %>% 
  map(~difference_in_means(listB ~ trt_B, data = .)) %>% 
  map(tidy) %>% 
  bind_rows(.id = "experiment")

pool = cali %>% 
  mutate(id = row_number()) %>% 
  pivot_longer(cols = c(listA, listB)) %>% 
  rename(list = name, count = value) %>% 
  mutate(Z = ifelse(trt_A == 1 & list == "listA" | trt_B == 1 & list == "listB",
                    1, 0)) %>% 
  split(.$experiment) %>% 
  map(~ lm_robust(count ~ Z + list, clusters = id, se_type = "stata", data = .)) %>% 
  map(tidy) %>% 
  bind_rows(.id = "experiment") %>% 
  filter(term == "Z")
  
est_df = rbind(a, b, pool)

est_df$term = fct_relevel(est_df$term, "trt_A", "trt_B", "Z")

Learning goals

  • Specific: Different versions of the same survey question change the answers we get

  • General: Illustrate bias-variance tradeoff in statistics

  • Context: List experiments

Bias-variance tradeoff as darts

This shows up in causal inference, machine learning, measurement, and many more!

Questions

  • Have you lied about having COVID symptoms?

  • Would you bribe a police officer to avoid a traffic ticket?

  • Have you had sex after drinking alcohol?

  • Have you been offered goods or favors for your vote?

  • Do you know anyone with links to a militant organization?

  • Would you oppose a black family moving next door?

  • Would you allow Muslim immigrants to become citizens?

What do these have in common?

  • They are sensitive questions

  • We can only learn about them using surveys

  • But asking about them directly leads to misreporting

  • This form of measurement error is called sensitivity bias

Techniques to reduce sensitivity bias

  • Honesty appeals

  • Communicating confidentiality protocols

  • Randomized response

  • Network scale-up

  • Endorsement experiments

  • List experiments

Techniques to reduce sensitivity bias

  • Honesty appeals

  • Communicating confidentiality protocols

  • Randomized response

  • Network scale-up

  • Endorsement experiments

  • List experiments

Example

List experiment

Here is a list of things that some people have done.

List experiment

Please listen to them and then tell me HOW MANY of them you have done in the past two years.

List experiment

Do not tell me which ones. Just tell me HOW MANY:

 

Control group

  1. Discussed politics with family or friends
  2. Cast a ballot for governor Phil Bryant
  3. Paid dues to a union
  4. Given money to a Tea Party candidate

List experiment

Do not tell me which ones. Just tell me HOW MANY:

 

Treatment group

  1. Discussed politics with family or friends
  2. Cast a ballot for governor Phil Bryant
  3. Paid dues to a union
  4. Given money to a Tea Party candidate
  5. Voted “YES” on the Personhood Initiative

Prevalence rate

\[ \text{Prevalence(Voted yes)} =\\ \text{Mean(List with sensitive item)} -\\ \text{Mean(List without sensitive item)} \]

  • Estimate: Proportion of individuals in the target population who hold the sensitive attitude or behavior

  • But we do not know how individual respondents voted!

Compare with direct question

Did you vote YES or NO on the Personhood Initiative, which appeared on the November 2011 Mississippi General Election Ballot?

\[ \text{Prevalence(Voted yes)} =\\ \text{Mean(Voted yes)} \]

Validation

# got this through
# https://automeris.io/WebPlotDigitizer/
# And by reverse engineering table 1
# Because replication data was not available
# So this has some human error
validation = data.frame(
  technique = c("Direct question", "List experiment"),
  estimate = c(0.396, 0.487),
  upper = c(0.419, 0.562),
  lower = c(0.376, 0.417)
)

actual_vote = 0.65

ggplot(validation) +
  aes(x = technique, y = estimate) +
  geom_point(size = 4, alpha = 0) + 
  geom_linerange(aes(x = technique, ymin = lower, ymax = upper),
                 linewidth = 1, alpha = 0) +
  ylim(0.2, 0.8) +
  labs(x = "Technique", 
       y = "Estimated proportion of 'No' votes")

Validation

ggplot(validation) +
  aes(x = technique, y = estimate) +
  geom_hline(yintercept = actual_vote, 
             linetype = "dashed",
             linewidth = 1) +
  geom_point(size = 4, alpha = 0) + 
  geom_linerange(aes(x = technique, ymin = lower, ymax = upper),
                 linewidth = 1, alpha = 0) +
  annotate("text", x = 0.7, y = 0.7, size = 5, 
           label = "Actual vote share") +
  ylim(0.2, 0.8) +
  labs(x = "Technique", 
       y = "Estimated proportion of 'No' votes")

Validation

ggplot(validation) +
  aes(x = technique, y = estimate) +
  geom_hline(yintercept = actual_vote, 
             linetype = "dashed",
             linewidth = 1) +
  geom_point(size = 4) + 
  geom_linerange(aes(x = technique, ymin = lower, ymax = upper),
                 linewidth = 1, alpha = 0) +
  annotate("text", x = 0.7, y = 0.7, size = 5, 
           label = "Actual vote share") +
  ylim(0.2, 0.8) +
  labs(x = "Technique", 
       y = "Estimated proportion of 'No' votes")

Validation

ggplot(validation) +
  aes(x = technique, y = estimate) +
  geom_hline(yintercept = actual_vote, 
             linetype = "dashed",
             linewidth = 1) +
  geom_point(size = 4) + 
  geom_linerange(aes(x = technique, ymin = lower, ymax = upper),
                 linewidth = 1) +
  annotate("text", x = 0.7, y = 0.7, size = 5, 
           label = "Actual vote share") +
  ylim(0.2, 0.8) +
  labs(x = "Technique", 
       y = "Estimated proportion of 'No' votes")

List experiments are not always an improvement

Can we do better?

Double list experiment

List A

  • Californians for Disability (advocating for people with disabilities)
  • California National Organization for Women (advocating for women’s equality and empowerment)
  • American Family Association (advocating for pro-family values)
  • American Red Cross (humanitarian organization)

List B

  • American Legion (veterans service organization)
  • Equality California (gay and lesbian advocacy organization)
  • Tea Party Patriots (conservative group supporting lower taxes and limited government)
  • Salvation Army (charitable organization)

Sensitive item

Organization X (advocating for immigration reduction and measures against undocumented immigration)

  • Randomly appears in list A or B

  • Single list: Half of the respondents see sensitive item

  • Double list: Everyone sees it

  • Equivalent to making two parallel list experiments

Three prevalence estimators

\[ \text{Prevalence}_A = \text{Mean}(A_t) - \text{Mean}(A_c) \]

\[ \text{Prevalence}_B = \text{Mean}(B_t) - \text{Mean}(B_c) \]

\[ \text{Prevalence}_{Pooled} = (\text{Prevalence}_A + \text{Prevalence}_B)/2 \]

DLE yields more precise estimates

ggplot(est_df %>% filter(experiment == "X")) +
  aes(x = term, y = estimate, shape = term) +
  geom_hline(yintercept = 0, linetype = "dashed") + 
  geom_point(size = 4, position = position_dodge(width = 0.5), alpha = 0) +
  geom_linerange(aes(x = term, ymin = conf.low, ymax = conf.high), 
                 size = 1, position = position_dodge(width = 0.5), alpha = 0) +
  theme(legend.position = "none") +
  labs(subtitle = "Organization X (immigration reduction)",
       x = "Estimator", 
       y = "Proportion support") +
  scale_x_discrete(labels = c("List A", "List B", "Pooled"))
Warning: Using `size` aesthetic for lines was deprecated in ggplot2 3.4.0.
ℹ Please use `linewidth` instead.

DLE yields more precise estimates

ggplot(est_df %>% filter(experiment == "X")) +
  aes(x = term, y = estimate, shape = term) +
  geom_hline(yintercept = 0, linetype = "dashed") + 
  geom_point(size = 4, position = position_dodge(width = 0.5)) +
  geom_linerange(aes(x = term, ymin = conf.low, ymax = conf.high), 
                 size = 1, position = position_dodge(width = 0.5)) +
  theme(legend.position = "none") +
  labs(subtitle = "Organization X (immigration reduction)", 
       x = "Estimator", 
       y = "Proportion support") +
  scale_x_discrete(labels = c("List A", "List B", "Pooled"))

But this is not for free

  • Need the two questions to measure the same thing

  • Advice: Pick baseline lists that are as similar as possible

  • BUT that makes it easier to spot the sensitive item

This can happen

ggplot(est_df %>% filter(experiment == "Y")) +
  aes(x = term, y = estimate, shape = term) +
  geom_hline(yintercept = 0, linetype = "dashed") + 
  geom_point(size = 4, position = position_dodge(width = 0.5), alpha = 0) +
  geom_linerange(aes(x = term, ymin = conf.low, ymax = conf.high), 
                 size = 1, position = position_dodge(width = 0.5), alpha = 0) +
  theme(legend.position = "none") +
  labs(subtitle = "Organization Y (citizen border patrol)", 
       x = "Estimator", 
       y = "Proportion support") +
  scale_x_discrete(labels = c("List A", "List B", "Pooled"))

This can happen

ggplot(est_df %>% filter(experiment == "Y")) +
  aes(x = term, y = estimate, shape = term) +
  geom_hline(yintercept = 0, linetype = "dashed") + 
  geom_point(size = 4, position = position_dodge(width = 0.5)) +
  geom_linerange(aes(x = term, ymin = conf.low, ymax = conf.high), 
                 size = 1, position = position_dodge(width = 0.5)) +
  theme(legend.position = "none") +
  labs(subtitle = "Organization Y (citizen border patrol)", 
       x = "Estimator", 
       y = "Proportion support") +
  scale_x_discrete(labels = c("List A", "List B", "Pooled"))

Summary

  • Three approaches to sensitive questions

  1. Direct question: High bias, high precision

  2. List experiment: Low bias, low precision

  3. Double list experiment: Low bias, high precision
    (but easier to break)

  • Need extensive piloting to choose well

Activity (groups 3-4)

  • Imagine something people tend to hide

  • How common do you think it is among your friends/peers/community?

  • Design a list experiment to learn about it

  • How many people would you ask?

  • How many lists?

  • How many and which baseline items?

  • Would it be any different from asking directly?

End product

Research overview

Bias-variance as darts revisited

Different players different winning strategies

Appendix

List experiment assumptions

1. No liars

Those who do not hold the sensitive item never falsely claim to bear it.

2. No design effects

Including the sensitive item does not change how participants respond to the baseline items

Types of sensitive items

  1. Frowned-upon attitude

    • “I would hate it if a black family moves next door”

  2. Pretend-to attitudes

    • “Of course I support the government!”

Types of DLE

designs = data.frame(
  expand.grid(
    Lists = c("Fixed", "Randomized"),
    Sensitive_item = c("Fixed", "Randomized")
  )
)

colnames(designs) = c("List order", "Sensitive item location")

designs %>% 
  kbl(booktabs = TRUE,
      escape = FALSE,
      align = "cc")
List order Sensitive item location
Fixed Fixed
Randomized Fixed
Fixed Randomized
Randomized Randomized

Carryover design effects

toy = data.frame(
  pot = c("Baseline", rep(c("\\(z_i = 1\\)", "\\(z_i = 0\\)"), 3)),
  Ya = c(2, 1, 2, 3, 2, 3, 2),
  Yb = c(2, 1, 1, 3, 3, 2, 3),
  zz = c(0, 1, -1, 1, -1, 1, -1)
)

toy = toy %>% 
  mutate(diff = Ya - Yb,
         sgn = diff * zz) %>% # (z_i - (1-z_i)) (Y_{i1} - Y_{i2})
  select(pot, Ya, Yb, diff)

toy = toy[1:5,]

colnames(toy) = c("Observed response",
                  "\\(Y_{i1}\\)",
                  "\\(Y_{i2}\\)",
                  "\\((Y_{i1} - Y_{i2})\\)")

toy %>% 
  kbl(escape = FALSE,
      align = "lcccc") %>% 
  pack_rows("Deflation", 2, 3) %>% 
  pack_rows("Inflation", 4, 5) %>% 
  column_spec(1, bold = c(T, F, F, F, F, F, F))
Warning in ensure_len_html(bold, nrows, "bold"): The number of provided values
in bold does not equal to the number of rows.
Observed response \(Y_{i1}\) \(Y_{i2}\) \((Y_{i1} - Y_{i2})\)
Baseline 2 2 0
Deflation
\(z_i = 1\) 1 1 0
\(z_i = 0\) 2 1 1
Inflation
\(z_i = 1\) 3 3 0
\(z_i = 0\) 2 3 -1

DLEs are costly to implement

load("cost_sim.rda")

cost_df = cost_sim %>% 
  mutate(rho = recode(as.factor(rho),
                      "0" = "rho == 0",
                      "0.4" = "rho == 0.4",
                      "0.8" = "rho == 0.8")) %>% 
  group_by(estimator, rho, cost) %>% 
  summarize(power = mean(p.value <= 0.05, na.rm = TRUE))
`summarise()` has grouped output by 'estimator', 'rho'. You can override using
the `.groups` argument.
# Reorder terms
cost_df$estimator = fct_relevel(cost_df$estimator,
                                "List A", "List B", "DLE")

ggplot(cost_df %>% filter(rho == "rho == 0" &
                            estimator != "List B")) +
  aes(x = cost, y = power, linetype = estimator, shape = estimator) + 
  geom_hline(yintercept = 0.8, linetype = "dotted") +
  labs(x = "Sample size loss",
       y = expression(paste('Power at ', alpha  == 0.05)),
       shape = "Design",
       linetype = "Design") +
  geom_line(size = 1) + geom_point(size = 2) +
  scale_y_continuous(limits = c(0, 1), breaks = seq(0, 1, by = 0.2)) +
  scale_shape_discrete(labels = c("Single list", "DLE")) +
  scale_linetype_discrete(labels = c("Single list", "DLE")) +
  theme(legend.position = "bottom")

Negatively correlated items

Things done in the last six months:

  1. I saw people playing sports in my neighborhood
  2. I visited friends on weekends
  3. I participated in activities organized by feminist groups
  4. I went to religious ceremonies at the church in my neighborhood

Negatively correlated items

Things done in the last six months:

  1. I saw people playing sports in my neighborhood
  2. I visited friends on weekends
  3. I participated in activities organized by feminist groups
  4. I went to religious ceremonies at the church in my neighborhood

Positively correlated lists

Items paired by number:

List A

  1. I saw people playing sports
  2. I visited friends
  3. I went to activities by feminist groups
  4. I went to religious ceremonies

List B

  1. I saw people playing soccer
  2. I chatted with my friends
  3. I went to activities by LGBTQ groups
  4. I donated to charity at church

Randomized response

For this question, I want you to answer yes or no.

Randomized response

For this question, I want you to answer yes or no. But I want you to consider the number of your dice throw.

Randomized response

For this question, I want you to answer yes or no. But I want you to consider the number of your dice throw. If shows on the dice, tell me no.

Randomized response

For this question, I want you to answer yes or no. But I want you to consider the number of your dice throw. If shows on the dice, tell me no. If shows, tell me yes.

Randomized response

For this question, I want you to answer yes or no. But I want you to consider the number of your dice throw. If shows on the dice, tell me no. If shows, tell me yes. But if another number shows, tell me your own opinion about the question.

 

[TURN AWAY FROM RESPONDENT]

 

Now you throw the dice so that I cannot see what comes out.

Randomized response

For this question, I want you to answer yes or no. But I want you to consider the number of your dice throw. If shows on the dice, tell me no. If shows, tell me yes. But if another number shows, tell me your own opinion about the question.

 

[TURN AWAY FROM RESPONDENT]

 

Have you thrown the dice?

Randomized response

For this question, I want you to answer yes or no. But I want you to consider the number of your dice throw. If shows on the dice, tell me no. If shows, tell me yes. But if another number shows, tell me your own opinion about the question.

 

[TURN AWAY FROM RESPONDENT]

 

Have you picked it up?

Randomized response

For this question, I want you to answer yes or no. But I want you to consider the number of your dice throw. If shows on the dice, tell me no. If shows, tell me yes. But if another number shows, tell me your own opinion about the question.

 

Now, during the height of the conflict in 2007 and 2008 (in Afghanistan), did you know any militants, like a family member, a friend, or someone you talked to on a regular basis?

Please, before you answer, take note of the number you rolled on the dice.